Repository navigation
Integrate Wave 2 gateway features - #25
Conversation
Provider catalogs can declare a model's precision (fp32, fp16, bf16, fp8, int8, int4) and quantization scheme. MODEL_PRECISION_PREFERENCE takes an ordered JSON list of precisions and is unset by default. While unset, catalog output and auto:intelligence candidate order are unchanged. When set, precision is recorded only from a provider's explicit catalog fields, never inferred from a model name. It breaks ties between candidates the quality and fast profiles already rank equally. It never adds a candidate, crosses a ranking group, reorders the balanced chain or changes an explicitly requested model. The intelligence policy can override the setting with its own precision_preference list. An invalid environment value is logged once and treated as unset.
RESPONSE_CACHE_POLICY_REVISION selects the response cache identity. legacy, the default (also when unset or empty), keeps today's keys, bodies and headers. v1 adds a canonical digest of the resolved route, the key's model permissions, its secret-scan mode, the retention policy and the workflow version. A policy change therefore misses instead of replaying an answer produced under different rules. Under v1, routes whose effective plan is not fully known (free pools, cascades, health-ordered auto routes, query parameters or provider-selection headers) bypass the cache. An answer is not stored if the policy changed while it was being generated. Unknown revisions bypass the cache, and old entries are never purged, only left unreachable.
scripts/probe_cost_forecast.py reads a JSON plan (from a file or stdin) describing the probe schedule, routes and prices. It prints upper bounds on provider calls and USD per day, per month and per run, and exits nonzero when a supplied --max-calls or --max-cost-usd limit would be exceeded. The script never runs probes or contacts providers, and does not read the application environment. A price the plan does not state is reported as null, never as zero.
GET /openapi.json returns an OpenAPI 3.1 description of the public client API: Chat Completions, Responses, Anthropic Messages and token counting, models and image generation, including the opencode and codex-easy paths. It also covers the documented MultiLLM headers and the error and SSE envelopes. It requires a dashboard login or an API key with the models scope. services/openapi_spec.py builds the spec without reading provider configuration, and docs/openapi.json is its committed output. A test fails when the two drift. Administrator routes are excluded.
scripts/model_capability_probe.py plans small capability probes (tools, JSON output, vision and similar) for one provider and model through the gateway. By default it only prints the plan. Sending requests needs both --execute and --allow-generation, known input and output prices, and stays within request, token, cost and time limits. The gateway key comes from an environment variable and is never printed. Each receipt records digests of the exact redacted configuration, fixtures and policy revision it measured. A result taken under a different configuration is therefore reported as stale instead of being reused.
A client that sends X-MultiLLM-Conversion-Report: 1 gets X-MultiLLM-Conversion-Fidelity and X-MultiLLM-Conversion-Fields response headers. They summarise which request and response fields a Chat, Responses or Messages translation kept, approximated, dropped or could not represent. The summary is content-free. Without the header, payloads and responses are unchanged. POST /v1/conversion/report returns the same report for a request without dispatching it upstream. It requires an API key, respects the key's model allowlist and limits the body to 1 MiB. The translated-protocol dispatch moves to routes/unified_bridge.py, keeping routes/unified.py under 1,000 lines. Both CORS lists allow the new request header and expose the report headers.
With SHADOW_EVAL_NOISE_FLOOR_ENABLED=true, an administrator can start a run with three_arm: true. Each comparison then replays the production model a second time, so judge noise is measured on identical answers, and a challenger's win counts only when it clears that floor under a seeded bootstrap. A claim atomically reserves all nine upstream calls (three arms, six judgments) against the daily cap; if the cap cannot cover them, nothing runs. The Worker D1 validator accepts the bounded noise_floor result only while the setting is on. scripts/replay_routing_cost.py replays stored samples and usage offline to compare what another route order would have cost. It reports unknown prices as null and never sends a request. With the setting off, sampling, judging, daily-cap accounting, stored rows and league output are unchanged.
Provider usage is now parsed into a typed observation whose input and output counts are each optional, with a provider, estimated or unknown basis. A missing count is no longer recorded as 0, and the ledger records cost_usd and cost_basis as null when a priced component is unknown. Embeddings have a measured output of 0. SSE usage is merged per component across events, so a stream that reports input and output counts in different events (such as Anthropic message_start and message_delta) records both. For budget limits only, a priced request with partial usage is charged conservatively, with missing components filled from the request estimate. Complete usage is accounted exactly as before, and response bodies pass through unchanged.
…rate/gateway-waves-1-2
…e/gateway-waves-1-2
…ntegrate/gateway-waves-1-2
…grate/gateway-waves-1-2
…egrate/gateway-waves-1-2
…ntegrate/gateway-waves-1-2
…o integrate/gateway-waves-1-2 # Conflicts: # package.json
…egrate/gateway-waves-1-2
An invalid MODEL_PRECISION_PREFERENCE is logged once and treated as unset, so the opening section no longer says it fails validation.
The Worker forwards RESPONSE_CACHE_POLICY_REVISION to the Container with its other environment settings.
The forecast doc now links the capability probe command's own execution budget instead of naming an internal work item.
…rate/gateway-waves-1-2
…e/gateway-waves-1-2
…ntegrate/gateway-waves-1-2
PROMPT_TEMPLATES_ENABLED (default off) adds admin-published, immutable template versions and a scoped render endpoint. Rendering substitutes declared {{name}} variables once, with bounded sizes and no code, file or network access. Templates are stored in the workbench SQLite database locally or in a new D1 control-state domain (migration 0014). With the flag on and the table missing, the routes return JSON 503.
…grate/gateway-waves-1-2 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json
CONFIG_SNAPSHOTS_ENABLED (default off) adds admin snapshots of auto-route candidate orders with a paginated diff and a confirmed apply. Apply runs as one D1 batch that checks the revision and stored route state, writes the routes, bumps the revision and appends an audit record, so a stale review returns 409. Rollback is a new snapshot. Snapshots use migration 0015 and fail closed with JSON 503 when its tables are missing.
…grate/gateway-waves-1-2 # Conflicts: # package.json
Adds an Opt-in gateway features section that links the document for each Wave 2 feature.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| 🔵 In progress View logs |
multillm-proxy | 6ed5e64 | Oct 09 2026, 10:41 AM |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Three tests started this machine's virtualenv Python by absolute path, so they failed on GitHub Actions once the dependency audit stopped masking them. The Worker tests now use PYTHON, then .venv, then python3, as the D1 integration test does, and the node job installs the locked Python dependencies.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
On a slow runner the 20 ms deadline expired before the request reached the fetcher, so the hold was released instead of kept as unknown. The fetcher never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the 5000 ms policy one; a 3000 ms bound still tells them apart.
…ails An intermittent CI failure returned 502 with no log, so the assertion now reports the provider call count and the response body.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
Integrate Wave 3 gateway features
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37552414 | Triggered | Generic Password | c8787fc | worker/alert-delivery.mjs | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Summary
This branch combines the Wave 1 integration branch (#14) with all ten Wave 2 feature branches, so they can be tested and merged together. It is based on
integrate/gateway-wave2, so this diff shows only Wave 2.MODEL_PRECISION_PREFERENCERESPONSE_CACHE_POLICY_REVISIONX-MultiLLM-Conversion-Report: 1request headerSHADOW_EVAL_NOISE_FLOOR_ENABLEDPROMPT_TEMPLATES_ENABLED, migration 0014CONFIG_SNAPSHOTS_ENABLED, migration 0015Integration changes
package.jsontest:workerlist;tests/fixtures/secret_firewall_coverage.json).Before merging to main
Two D1 migrations are included:
0014_prompt_templates.sqland0015_config_snapshots.sql. A push to main does not apply them, and/readyreportsd1_schema_missinguntil they exist. Apply them first:Tests
npm run test:worker: 1,227 passed.cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.